---
sidebar_label: Best Practices
description: Best practices for OpenTelemetry of Hasura GraphQL Engine
title: 'OpenTelemetry Best Practices'
keywords:
  - hasura
  - docs
  - enterprise
  - opentelemetry
  - open telemetry
  - traces
  - metrics
  - logs
sidebar_position: 4
---

import ProductBadge from '@site/src/components/ProductBadge';

# OpenTelemetry Best Practices

<div className="badge-container">
  <ProductBadge free pro ee self />
</div>

## Identify different applications

GraphQL Engine uses the `hasura` service name by default. If you have many different GraphQL Engine applications you
should configure different `service.name` attributes for each application to identify and filter metrics more easily.

## Use OpenTelemetry Collector

Most of observability services require the
[OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector) as a proxy to push data. Using
OpenTelemetry Collector also helps:

- Route and export data to multiple services.
- Optimize the cost with sampling and filter processors.
- Transform and calculate aggregate metrics.

## Cost optimization

### Sampling

The OpenTelemetry Collector supports many strategies to sample spans and log records. See OpenTelemetry Collector Contrib
docs for more context.

- [Probabilistic Sampling Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/probabilisticsamplerprocessor).
- [Tail Sampling Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor).

### Trace

You may not want to trace all operations, for example, you only want to trace GraphQL requests or individual GraphQL
operations. Let's configure the
[Filter Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/filterprocessor).

```yaml
processors:
  filter:
    error_mode: ignore
    traces:
      span:
        # - 'IsMatch(name, "Event trigger")'
        # - 'IsMatch(name, "Scheduled trigger")'
        # - 'IsMatch(name, "websocket")'
        - 'IsMatch(name, "/v1/version")'
        - 'IsMatch(name, "/v1/entitlement")'
        - 'IsMatch(name, "/v1alpha1/config")'
        # filter unused graphql operation
        - attributes["graphql.operation.name"] == "MyQuery"
```

### Logs

Similar to traces, besides disabling log types via
[environment variables](/observability/opentelemetry/graphql-engine.mdx#configurations), you may want to filter
individual logs:

```yaml
processors:
  filter:
    error_mode: ignore
    logs:
      log_record:
        - 'attributes["type"] == "query-log" and IsMatch(body["query"]["operationName"], "UnknownQuery")'
        - 'attributes["type"] == "http-log" and IsMatch(body["operation"]["query"]["operationName"], "UnknownQuery")'
```

Check out more configuration examples [here](https://github.com/hasura/graphql-engine/blob/master/community/boilerplates/observability/enterprise/otel-collector/otel-collector-config.yaml).

## Monitoring

### Logging

OpenTelemetry logs are printed in an `unstructured` type with a `warn` level if runtime errors happen.

```json
{
  "detail": "OTel exporter: Failed to deliver logs: Encountered retryable HTTP exception: ConnectionFailure ...",
  "level": "warn",
  "timestamp": "2024-06-19T06:39:51.704+0000",
  "type": "unstructured"
}
```

### Metrics

GraphQL Engine exports metrics to monitor the number of sent and dropped (failed) trace spans or log records. You can
collect them via OpenTelemetry metrics or the native Prometheus exporter.

See the list of available metrics
[here](/observability/enterprise-edition/prometheus/metrics.mdx#opentelemetry-otlp-export-metrics).

Download and install [the Grafana dashboard](https://grafana.com/grafana/dashboards/21363) if you are using Prometheus.
